Papers with pretraining techniques
Improving Pretraining Techniques for Code-Switched NLP (2023.acl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models for code-switched inputs are a key component of NLP applications. |
| Approach: | They propose to use masked language modeling techniques to mask code-switched text that are cognizant of language boundaries prior to masking. |
| Outcome: | The proposed techniques improve performance on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), compared to standard pretraining techniques. |
Larger-Context Tagging: When and Why Does It Work? (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing tagging systems that use sentence-level data are not well understood. |
| Approach: | They propose a larger-context approach to tagging tasks that incorporates contextual information into existing tapping systems. |
| Outcome: | The proposed aggregators improve on four tagging tasks and 13 datasets. |
Pretraining Language Models for Diachronic Linguistic Change Discovery (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used as knowledge discovery tools . historical linguistics and literary studies often construct arguments on the basis of distinctions between phenomena like time-period or genre. |
| Approach: | They propose to use LLMs to train large language models over modest historical corpora without allowing contamination from anachronistic data. |
| Outcome: | The proposed model better respects historical divisions and is more computationally efficient compared to the standard approach of fine-tuning an existing LLM. |